Artificial Neural Networks (ANNs), or Neural Networks (NNs) for short, are algorithmic mathematical models imitating the behavior characteristics of animal neural networks and performing the distributed concurrent information processing. Depending on the complexity of a system, such networks adjust interconnection among a great number of internal nodes, thereby achieving the purpose of information processing. The algorithm used by NNs may be vector multiplication (also referred to as “multiplication”) and convolution, which widely adopts sign functions and various approximations thereof.
As neural networks in animal brains, NNs consist of multiple interconnected nodes. As shown in FIG. 3, each block represents a node and each arrow represents a connection between two nodes.
The calculation formula of a neuron can be briefly described as
      y    =          f      ⁡              (                              ∑                          i              =              0                        n                    ⁢                                          ⁢                                    w              i                        *                          x              i                                      )              ,wherein x represents input data received at all input nodes connected to the output nodes, w represents corresponding weight values between the input nodes and the output nodes, and f(x) is a nonlinear function, usually known as an activation function including those commonly used functions such as
      1          1      +              e                  -          x                      ⁢          ⁢  and  ⁢          ⁢                              e          x                -                  e                      -            x                                                e          x                +                  e                      -            x                                .  
NNs are widely applied to a variety of applications, such as computer vision, voice recognition and natural language processing. In recent years, the scale of NNs has been growing. For example, in 1998, Lecun's neural network for handwriting characters recognition includes less than 1M weight values; while in 2012, Krizhevsky for participating ImageNet competition includes 60M weight values.
As deep learning technology develops, the scale of the current neural network and the amount of neural network data are increasing, which imposes growing demands on storage performance and memory access bandwidth.